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ABSTRACT 

This document presents the final report on the 
evaluation of California's model student assistance program (SAP) 
demonstration projects implemented in five locations across the state 
from July 1989 through June 1992. The report provides an overall, 
integrated review of the evaluation of the SAP demonstration 
projects, summarizes important findings from previous evaluation 
phases, reports evaluation activities for the project's third year, 
and addresses four questions: (1) how successfully did the SAPs 
develop their administrative structures and organization? (2) who 
were the students who participated in the SAPs? (3) how effectively 
did the SAPs deliver services to students? and (4) did the SAPs have 
a positive impact on participating students? Section 1 of the report 
provides an introduction to the SAP project. Section 2 presents a 
review of the political and organizational development of the 
demonstration projects. Section 3 provides a summary of the 1990-91 
process evaluation. Section 4 details the third-year evaluation 
effort which was dedicated to the evaluation of the impact on 
students of the SAP demonstration projects. Key issues include the 
development of the evaluation methodology, the analysis of data, and 
the interpretation of results. Section 5 includes an integration of 
all 3 years of evaluation findings, providing some of the insights 
and recommendations regarding the demonstration projects and the 
functioning of SAPs. (NB) 
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SECTION 1: INTRODUCTION 



This is the final report on the evaluation of the model student assistance program (SAP) 
demonstration projects. These projects, funded by the California Department of Alcohol and Drug 
Programs (DADP), were implemented in five locations: Clovis Unified School District, Mariposa 
County Unified School District, a consortium of four unified school districts in Mendocino County 
(led by the Mendocino County Office of Education), Pajaro Valley Unified School District, and the 
San Dieguito Union High School District. The funding for these projects began in July 1989 and 
continued through June 1992. 

Purposes of the Final Report 

First, this report provides an overall, integrated review of the evaluation of the SAP demonstration 
projects. This has been a long, and complex, evaluation. It is important to provide an overview of 
the development of the evaluation and the overall strategy that has guided its conduct. 

Second, this report summarizes some of the most important findings from previous phases of 
the evaluation. For complete findings, and in those instances where more detailed findings are 
helpful, the reader is referred to the previous evaluation reports. 

Third, the evaluation activities for the third year are reported in detail. The third year of the 
evaluation was the Outcome evaluation. The key activities in this year included developing the 
measurement instruments, forming a control group, and assessing the impact of SAP activities on 
the students. 

Finally, this report addresses four general questions: 

1 . How successfully did the S APs develop their administrative structures and 
organization? 

2. Who were the students who participated in the SAPs? Did the participating students 
represent a cross section of the student body, or were the participating students 
somehow a specialized subpopulation? 

3 . How effectively did the SAPs deliver services to students? What kinds of services 
were delivered? Were those services appropriate given the students* life situations? 

4. Did the SAPs have a positive impact on participating students? 
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Organization of the Final Report 

The organization of the final report reflects the purposes of the report as outlined above. In Section 
2, a review of the organization and development of the demonstration projects are presented. This 
section draws upon work done in the evaluation process. 

Section 3 provides a summary of the 1990-91 Process evaluation. The Process evaluation 
findings played a significant role in shaping the conduct of this year's evaluation efforts. The 
Process evaluation was quite iirformative about the functioning and organization of the projects. 

Section 4 details the third-year evaluation effort which was dedicated to the evaluation of the 
impact on the students of the SAP demonstration projects. This section represents the heart of the 
evaluation project Key issues in this section are the development of the evaluation methodology, 
the analysis of the collected data, and the interpretation of the results. To the best of SWRL's 
knowledge, this evaluation is the first conducted of a SAP using an experimental approach. 

Section 5 includes an integration of all three years of evaluation findings. This section goes 
beyond a strict interpretation of the data presented in Section 4 and provides some of the insights 
and recommendations regarding the demonstration projects and the functioning of S APs. 

Limitations of the Final Report 

This evaluation has made a significant contribution to answering many of the important questions 
regarding the effectiveness of S APs. This report provides evidence that SAPs contribute to a 
reduced level of alcohol and drug use, and can help students with several important 
social/psychological issues. Unfortunately, there are other student outcomes (e.g., improvements 
in family functioning) that are of interest but which could not be answered by this evaluation. 

Also, there is always the need for replication of findings. Any single study, because it could 
be taking advantage of chance or unusual occurrences, can provide unreliable results. This is true 
in this evaluation. These projects were selected because they were the best possible projects. The 
projects were managed by highly motivated, creative, and effective individuals. If S APs were 
more widely disseminated across California, the quality of management would not be as uniformly 
high. Thus, it is conceivable that the quality of work found in these projects would not be matched 
with highly disseminated projects. 

It also should be noted that throughout this evaluation, S WRL has not emphasized differences 
among the projects. Instead, site differences have been minimized in previous reports, while 
commonalties have been accentuated. This strategy has worked well to ensure the maximum 
amount of trust during the evaluation. Because of substantial overlaps among the projects in terms 
of their operations and structures, there is probably more to learn from an analysis of the 



commonalties of the projects than from their differences. However, differences related to 
significant findings are noted in this report. 
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SECTION 2: THE POLITICAL AND ORGANIZATIONAL 
DEVELOPMENT OF THE PROJECTS 



The Communities 

This section provides a short analysis of the demographic, economic, social and political 
characteristics of the cities/counties hosting the SAP demonstration projects. Tables 1-3 provide a 
summary of much of the information. 

Clovis 

Clovis is a medium-sized, rapidly growing suburb of about 50,000 people north of Fresno. The 
Clevis Unified School District (USD) covers a large geographic area, including many 
unincorporated areas of Fresno County. However, most of the students live within the city limits. 
Clovis was one of the two most economically affluent SAP sites. 

There were 23,856 students (K-12) in Clovis USD during the 1991-92 school year. There are 
two high schools, two intermediate schools, and one continuation school in the district; all of these 
schools participated in the SAP project. While the district is generally affluent, there are distinct 
socioeconomic differences between the two high schools and between the two middle schools. 
One of the high schools and one of the middle schools are located in an area with a distinctly higher 
socioeconomic status than their counterparts. The contrast is that of an upper-middle class 
population vs. a middle-class population; none of the schools could be described as having a large 
population of poor children. However, the differences among the schools were reported to create 
some competition and stresses within the district. 

Mariposa County 

Mariposa County is a small, sparsely populated county located about one hour's drive northeast of 
Fresno. Mariposa is rural, and many of the families in the county are poor. The school system in 
Mariposa is organized through the county schools. There are no other school districts in Mariposa 
County, and there are no incorporated cities. 

The primary industries in the county are tourism and agriculture. There are reports of a 
significant amount of illegal drug production and drug use in the more mountainous, isolated 
sections. Reportedly, drug production consists primarily of methamphetamine and marijuana 
cultivation. 
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Mariposa County had 2,452 (K-12) students during the 1991-92 school year. There is one 

■ high school, with approximately 450 students, and one middle school with approximately 225 
students. The elementary schools that feed into the middle and high schools are quite isolated. 

£ The high school and middle school are both located in the community of Mariposa (the city is not 

™ incorporated), which has about 3,500 people. 

m Mendocino County 

■ The Mendocino County SAP was organized as a consortium among four districts in the County: 
Ukiah USD, Willits USD, Mendocino USD, and Fort Bragg USD. Mendocino County is 

I generally rural and economically depressed. Unemployment and other economic indicators 

consistently show that the county performs worse than California as a whole. In fact, the 
g[ percentage of unemployment often is double that of the California average. During the entire 1989- 

■ 92 time span of the SAP demonstration project, the county suffcied economically. 

- Ukiah, which is the county seat, is the largest community. Ukiah USD has the largest high 

\ school by far, with 1,786 students in the 1991-92 school year. The other three districts, and their 

high schools, are significantly smaller, averaging only 488 high school students in the last school 

■ year. The smaller communities are geographically somewhat isolated from the rest of the county, 
with trips of an hour or more to get to Ukiah. This geographic isolation had a significant impact on 

M the development and structure of the SAP. It required that the SAP be developed essentially as 

™ four parallel programs, which operated fairly independently, although they are all managed through 

g| the Mendocino County Youth Project (MCYP). MCYP is a private, nonprofit organization that 

■ was subcontracted by the Mendocino County Office of Education to manage the SAP. 

Mendocino County is primarily white, with about 15% of the population being Hispanic. 

■ There is a small percentage of Native Americans (approximately 4%). There were virtually no 
blacks or Asians. As might be expected from the poor economic conditions in the county, there is 

■ significant evidence of 'ow family incomes for the students. About 45% of the students received 
free or subsidized lunches; about 20% of the families with children enrolled in the high schools 

£ received AFDC. 

M Pajaro Valley 

Pajaro Valley Unified School District is located south of Santa Cruz. It serves the communities of 

■ Watsonville and Aptos. There are distinct socioeconomic differences between Watsonville and 
Aptos. Watsonville is significantly less affluent and has a large Hispanic population. About fifty- 
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six percent of the population are Hispanic. About 33% of the students have limited English 
proficiency (LEP). Economically, the area is somewhat depressed. The economy in the local area 
is based on tourism and agriculture. Aptos, which is not an incorporated city, is about one fifth the 
size of Watsonville (8,000 vs. 32,000 residents) and is best characterized as a upper-middle class 
bedroom community that has a much smaller proportion of minority residents. Economically, 
Aptos appears prosperous. From information provided by key informants, Aptos often does not 
see itself as being a part of the greater Watsonville community despite the fact that it is linked to 
Watsonville through the school system. 

San Dieguito 

San Dieguito Union High School District is about 20 miles north of San Diego along Interstate 5 in 
the community of Encinitas. This area is fairly affluent and is largely white. There is a growing 
Hispanic and Asian community in the area. Many of the students have parents who commute into 
San Diego for employment. Although Encinitas has its own distinct identity, it is rapidly becoming 
a suburban community of San Diego. This district is probably the most affluent of all of the SAP 
sites. 

The Organization of the SAPs 

This section provides an overview of the basic structure and organization of the SAP 
demonstration projects. There was substantial variation in how the programs were organized and 
implemented. The variations in structure and operation can, in many cases, be traced to the social 
and political characteristics of the SAP settings. Table 2 summarizes the organizational 
characteristics of the SAPs. 

Clovis 

Clovis organized and implemented its SAP throughout the district, in all three high schools and 
both middle schools. The program began in spring 1989, with a pilot program instituted at Clovis 
High School. Because of the pilot program's success, the decision was made to apply for funding 
through D ADP. Except for Mariposa, Clovis had the least amount of programmatic experience 
providing "SAP-like" services to students prior to the SAP startup in September 1989. 

The SAP was structured using two administrative layers. First, there was a district core team. 
The district core team contained members from all major stakeholder groups in the district: 
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teachers, classified staff, site administrators, district administrators, one school board member, law 
enforcement, community-based organizations (CBOs), and parents. The district core team was 
responsible for the overall policy and management of the SAP. 

At the school-site level, school-site core teams were organized to coordinate the delivery of 
services provided to students. Representation on the school-site core teams emphasized personnel 
drawn from that school, such as teachers, site administrators, and school counselors. 

The balance in functions and representation between the district core and school-site core teams 
seemed to work well. Hie representatives at the district level served to tie the SAP into the broader 
community network, facilitating referrals of students both into and out of the program. The 
representatives at the school-site level provided a strong base for identification and referral of 
students at the school. Evidence for this comes from the fact that, based on 1990-91 Process 
evaluation statistics, the Clovis SAP (along with San Dieguito) had the broadest mix of referral 
sources of any of the SAP projects. 

To fund its SAP, Clovis USD combined Drug, Alcohol and Tobacco Education PATE) funds 
and a small supplement provided by the district itself, along with the money provided by D ADP. 
In fact, DADP funds represented only about 35% of the actual cost of the SAP. The funds were 
commingled, and it was meaningless to try and identify what program activities were supported by 
what specific monies. 

Mariposa County 

The SAP in Mariposa County began functioning in September 1989. Unlike the other SAP sites, 
Mariposa had never provided any "SAP-type" services to its students prior to DADP funding. The 
Mariposa SAP always functioned using a counselor-based model. A counselor-based model 
emphasizes the role of the SAP counselor taking full responsibility for the assessment of students 
and the delivery of services without the use of a core team to support or perform any of these 
activities. At this site, because of the small size of the school district, the SAP director (Marilyn 
North for the first year of the project and Kathy Bloom for the remaining two years) functioned 
both as the director and as the primary service provider in the SAP. An assistant also provided 
direct services to students. 

Because of the small size of the program, most referrals were made directly to the SAP 
director. The majority of referrals were self-referrals by the students, and the proportion of self- 
referrals increased steadily after the beginning of the program. There was no formal school-site 
core team to facilitate referrals or assessment of students. However, there was an informal 
network of school staff that were reported to be sensitive to students and helpful in getting the 
students in touch with the SAP. 
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A unique feature of the Mariposa County SAP is that the SAP functioned as a de facto county 
social service agency. Because of the general lack of services for much of the county, and because 
schools in Mariposa are organized at the county level, the SAP, since its inception, served as an 
alternative service delivery system fcr the mental health needs of children and families in the 
county. The SAP director routinely handled service calls from Child Protective Services and 
Mental Health, intervening on specific cases on behalf of the county. Usually this occurred when 
it was logistically difficult for the county worker to reach the case and it was more accessible to the 
SAP staff. In return, the county worked cooperatively with the SAP when asked. For example, in 
some cases when a student had a serious psychological dysfunction, the county worker secured 
appropriate hospitalization in Fresno County since Mariposa does not have a psychiatric hospital 
bed. The county, however, did not provide any funds to the SAP. 

The physical facilities of the SAP consisted of a small office at the high school and a small 
office at the middle school. The offices were satisfactory for conducting all SAP activities and for 
ensuring confidentiality of the services. This is mentioned because the SAP director, at the 
beginning of the project in September 1989, had difficulty securing any physical space at the high 
school for the SAP. The reasons for these difficulties are discussed later in this section. 

Mendocino County 

The Mendocino County SAP was developed and implemented through a joint effort of the county 
Division of Alcohol and Other Drug Programs, the four participating school districts (which are the 
four largest districts in Mendocino County), and the MCYP. As mentioned earlier, MCYP is a 
private, nonprofit organization that was subcontracted by the Mendocino County Office of 
Education to manage the SAP. The county office served as the lead administrative agency for the 
project. It, in turn, subcontracted with MCYP to hire SAP personnel and provide services to each 
of the high schools in the four participating districts. This organizational structure has been used 
many other times in Mendocino County to provide both in-school and out-of-school services to 
youth. MCYP actually operates as a special project of the Mendocino County Office of Education 
and has staff serving in other special programs implemented at all high schools and middle school 
campuses in the county. In effect, the SAP demonstration project supplemented existing MCYP 
activities. 

The MCYP SAP staff developed a core team model at each of the high schools. The core 
team's responsibilities included taking a lead role in maintaining, evaluating, refining, and 
expanding the SAP. This included the development of an effective and efficient referral process. 
The core teams varied somewhat across high schools, but generally included teachers, classified 
staff, site administrators, parents, community representatives, as well as the SAP counselor at that 
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high school. Much of the core team was informally in place at the high schools prior to the start of 
the SAP demonstration project because of the myriad of other activities that MCYP staff engage in 
at the school. 

Pajaro Valley 

A counselor-based model was implemented at Pajaro Valley Unified School District This model 
evolved into its current form over the five years prior to the start of the DADP funding. In this 
model, each school-site had one or more SAP counselors who accepted referrals of students with 
problematic behavior from school staff. The SAP counselor was responsible for devising an 
appropriate strategy for serving the student. Unlike a core team model, where there is input from 
several individuals who are collectively responsible for formulating a service plan for the student, 
the SAP counselor assumed most or all of this responsibility. 

Pajaro Valley relied heavily on interns to fill the SAP counselor positions at the school-sites. 
The interns were attracted to Pajaro because of the amount of training they received, the paid 
stipends, and the amount of supervision provided. 

The SAP demonstration program was only one of several Pajaro Valley programs serving the 
psychological and social needs of students. An elementary SAP, known as "Kids Corner," was 
implemented in 1989. Several other programs focused on the prevention of tobacco use, family 
services, and special needs for Hispanics, that were managed out of the same office managing the 
SAP. 

In July 1991, the SAP became a private, nonprofit agency that took over most of the activities 
carried out by these programs. The rationale for moving these projects out of the school district 
and into a private, nonprofit agency was that it provided greater freedom, flexibility, and funding 
opportunities. This was done on a friendly, cooperative basis with the school district. 

San Dieguito 

San Dieguito adopted a classic core team model for its SAP project. In many ways, San Dieguito 
was the typical textbook example of a structure for a SAP. The SAP was implemented at each of 
the middle and high schools in the district. The SAP developed, at most school-sites, into a stable 
and comprehensive system. In the San Dieguito system, each student who was identified for any 
unacceptable behavior was referred to the assistant principal. The assistant principal, in turn, 
decided whether the referral needed only disciplinary action or whether the student should be 
refeiTed to Student Assistant Services/Student Study Team (S AS/SST) located at that school-site. 
The SAS/SST group was responsible for conducting a comprehensive review of all information 
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that could be gathered as to why the student was not functioning well in school After review by 
the SAS/SST, the student could be referred to special education classes, or to SAP services, or 
another appropriate referral could be made. In effect, the SAS/SST was organized to act as a 
funnel through which all students exhibiting difficulties at school must pass. There was a 
conscious effort to make use of the SAS/SST structure to increase the integration of all student 
services in the district 

Once referred to the SAP by the SAS/SST, ihe student was assessed by the SAP counselor at 
the school-site to determine what were the most appropriate SAP services for the student. A 
variety of options, including on-campus groups, individual counseling, and referral to outside 
service providers, were available at that point. Specific procedures ensured that parents became 
appropriately involved with the student's services, and the teacher(s) who initially referred the 
student were provided reasonable feedback* In addition, the SAS/SST systematically conducted a 
follow-up on the progress of the student. 

The Social/Politics Context of the Program 

This section provides a review of the social/political context of each of the SAP programs. As 
mentioned earlier", there is evidence that, in many cases, the social/poetical context of the program 
has played a major role in shaping the structure and organization of the program. This section 
looks at how the social/political landscape has influenced project implementation. Table 3 
summarizes the information. 

Clovis 

The Clovis SAP had good political support within the district for three reasons. First, the Clovis 
SAP began its development with a strong philosophical statement supporting the need for services 
to students who were at risk of school failure because of social or psychological problems. The 
district formally recognized that a student's dysfunctional personal life can seriously hamper his or 
her academic performance. This argument was compelling in Clovis because this district has 
traditionally emphasized norms supporting strong academic achievement. 

A second reason for the strong political support for the SAP in this district was the 
demonstrated need for services for students, which was found in the pilot SAP developed at Clovis 
High School. The pilot rapidly identified many students who benefited from SAP services. This 
district reportedly has a history of denial of widespread alcohol and other drug (AOD)/ 
psychological problems. The rapid growth of the pilot project did much to break through that 
denial and cement district support. 
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However, perhaps the most important reason for the strong political support for the SAP in the 
district was due to the inclusion of numerous key school and community "players" in either an 
advisory role for the SAP or in direct service to the SAP through the district core team. Clovis 
appeared to be a district where decisions regarding school policy or operations could become 
quickly politicized Often times decisions are made only after consensus has built for a potential 
solution. Thus, the creation of a strong network of individuals within the district supporting the 
SAP was probably crucial for the long-term success of the project. In short, the political base of 
support for the SAP was built through means that were consistent with the norms of decision 
making that prevail in Clovis. 

Mariposa 

Of all the SAP demonstration projects, Mariposa had the weakest political support. SWRL 
believes this was due to three reasons. First, during the 1988-90 time frame, when the program 
was being formed, there were two strong factions vying for control of the school district. One 
faction was led by a majority of the 1988-89 school board members and by the county 
superintendent, who had been hired by the school board in 1988. In 1990 these same school 
board members were forced into a recall election and the superintendent was running in a regularly 
scheduled electro The opposing faction was generally led by more conservative county residents 
aligned with Mariposa County's continuation school principal, who was running against the 
incumbent for the superintendency. The net effect was that for two years the political events in the 
district forced every key player in the county to choose sides (Pollard, 1991). Throughout most of 
the crisis, the sides were relatively evenly divided, creating a political and administrative paralysis 
in the district. Virtually no changes in district operations and no major new initiatives in services 
provided to students (such as the implementation of the SAP) received attention from any 
administrators. 

The second complication in the political history of the Mariposa SAP was that its director 
(North) created the SAP from scratch in early 1989 (prior to the election) by approaching the old 
superintendent and asking if she could write a proposal in response to DADP's RFP. The 
superintendent was enthusiastic and the proposal was written. However, in September 1989, 
when North went to the high school to begin implementing the SAP, the principal of the high 
school did not know who she was, what the SAP program was, or what she was planning to do at 
the high school Because the high school principal was politically aligned with the conservative 
faction, the principal and North did not communicate well, and the principal had never been 
informed that the SAP proposal had been accepted and funded. This resulted in a substantial 
amount of resistance from the principal to the SAP. 



These organizational conflicts would cause problems for most individuals trying to implement a 
new program. North handled the issues by operating independently within the school system. 
She reported directly to the old superintendent and not to the high school principal. She operated 
in the high school by making informal alliances with teachers and staff who would refer students 
for services. She also operated much of the SAP outside of the high school grounds by working 
with students and their families in other settings such as their homes. North had the skills and 
temperament to operate in this fashion, and the ability to continually negotiate the political white 
water in the county. Otherwise, there is a good chance that this SAP would never have been 
implemented However, this strategy created a third problem for the SAP. This strategy prevented 
the SAP from becoming institutionalized in the school. 

Mendocino County 

SAP services in Mendocino County have strong political support. However, it is not the SAP 
demonstration project, per se, that is strongly supported. Instead, it is the MCYP, the 
organization within which the SAP is embedded, that has developed strong, and probably 
unbreakable, ties between itself and Mendocino County. Even though no further funding has been 
obtained for the SAP, SAP-like services will still be conducted under one or another of the projects 
being operated under the auspices of MCYP. This does not mean that the lack of funding will have 
no impact on services. There will probably be a significant reduction in the quantity of services 
delivered to youth in Mendocino County; many youth, however, still received some services if 
they qualified under one of the MCYP programs . 

The MCYP functions as a quasi-governmental agency in Mendocino County. Ongoing 
interaction between the county and MCYP occurs at a variety of levels. MCYP is involved in 
almost every county setting in which youth are involved, including juvenile justice settings. 
MCYP services have become essential to the human services network of the county. In large part 
this is due to the poor economic conditions that are so prevalent in this county. Because of the 
poor economy, the county simply has never been able to fund a traditional model for social 
services for youth to the extent that would be desired. Because of its flexibility and its ability to 
attract external funding on an ongoing basis, MCYP has picked up services where the county 
normally would be the only major player. It appears that these circumstances enhanced the political 
stability and support for MCYP. 

Because of these considerations, Mendocino County has not offered many significant insights 
into the political/social features that facilitate the adoption of a SAP. The SAP, once funded, was 
never subjected to the kinds of political stresses that new programs being implemented in a school 
setting typically encounter. The most important lesson may be that by "piggy backing' 1 the SAP 
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onto already established service delivery mechanisms, the SAP essentially had a painless birth. 
However, in school settings without this kind of opportunity, the lessons drawn from Mendocino 
County do not offer much guidance. 

Pajaro Valley 

Like Mendocino County, many "SAP-like" services were delivered in Pajaro Valley USD prior to 
the funding of the SAP demonstration project. More services have been added since the 
demonstration project began in 1989. As a result, the crucial political/social lessons regarding the 
implementation of SAPs cannot be drawn from 1989 onward The most important lessons should 
be drawn from the political and social issues surrounding the creation of the organization in 1984. 

The most important feature of the political/social context of Pajaro Valley is that much of the 
program's political support came directly from the local community separate from the formal 
school district organization. Nowhere is this better illustrated than in the strong support Pajaro 
Valley's programs received from parents. This support existed, in large part, because Pajaro 
Valley was the most successful of the five SAPs delivering services to parents. While most SAPs 
(including SAPs in California not participating in the demonstration project) complain that it is 
difficult to attract parent participation in the SAP, this was not the case for Pajaro Valley. For 
example, one member of the SWRL evaluation staff (Ann Bickel) observed the Pajaro Valley 
director (Linda Perez) hosting the fourth night of a four-night program for families. There were 
about 200 family members voluntarily participating. This is a far higher turnout than other school- 
based parenting or family programs are able to produce. 

An important question is how Pajaro Valley was able to generate such parent participation. The 
director says that, in large part, the parent participation was a result of several years of 
communicating with and involving parents. For example, six years ago the director of the SAP 
spent an entire year meeting with parents in their homes, in canneries, and wherever else they 
could be found, asking them what they would like to see the schools offer students. This kind of 
long-term development is unusual, to say the least. In most California school districts (particularly 
in today's budget climate) there is neither the time, nor the personnel, who can effectively tackle 
this task. In another effort, the SAP director created the "Safe Homes" program. This was an 
effort involving law enforcement, which had parents in the community sign a pledge that they 
would not allow alcohol to be served at their children's parties and that they would provide a 
chaperone on these occasions. Currently, 500 families are signed up in this project. 

While the above work represents building support from the grassroots, the program also 
effectively worked to generate support "top down." In 1984, just as the program was being 
organized, the director formed the "Committee of 25." This group consisted of two school board 
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members, teachers, site and district administrators, parents, and community leaders who worked to 
develop a plan for providing services to students in the community. This group helped to build a 
network of the key players in the community and district who would support the program. 

Just as important, the director provided 30 hours of training for almost all district personnel in 
1984-85, the first two years of the program's existence. This included special "study sessions" 
developed for school board members and senior district administrations. The director believed that 
the training helped sensitize these key individuals to the needs of students and the community and 
what the program could do to effectively and cost-efficiendy provide for those needs. The director 
was so effective in training the key players in the community that one school board member was 
elected on the basis of the AOD-related needs of the district. 

Community support also was developed in Pajaro Valley through an effective use of local 
media. There were literally dozens of stories that were reported in the local press regarding 
different aspects of the SAP's operations. The local media effectively "sold" the program to the 
community. This, of course, always has been an additional channel for reaching out to parents. 

In sum, the most accurate description of the political success of this project is that it patiently 
and carefully built support for itself among key community groups. The careful and thorough 
effort put into contacting parents, training school personnel, and working with the media (among 
other things), paid enormous dividends to the program in terms of district support. Most 
importantly, parents were involved in the SAPs services at much higher rates than was observed 
in any other SAP program known to SWRL. In turn, this resulted in strong parent support for the 
program, which eventually created strong district administrative support In effect, the project's 
support was built both top down and bottom up. 

San Dieguito 

The SAP program in San Dieguito, like the program in Pajaro Valley, was funded by DADP well 
after the program actually began serving students. And, as with Pajaro Valley, the important 
lessons come not from the implementation of the SAP since 1989, but from the implementation of 
the SAP since the beginning of the program's operations, in about 1985. 

Like the Clovis, Mendocino County, and Pajaro Valley SAPs, the SAP in San Dieguito 
enjoyed considerable political support from within the district. First, the project early in its 
development, had a clear and unambiguous policy statement recognizing that students 1 AOD use 
was a detriment to their school achievement. The director in San Dieguito also provided a 
significant amount of training in AOD-related issues for all district personnel. This sensitized key 
players in the district to the need for services and also demonstrated how implementation of the 
SAP could benefit the students and the district. Finally, the director involved a wide spectrum of 
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district personnel in core teams. This led to a solid integration of the program into the districts 
operations. 

The most interesting lessons to be learned from San Dieguito come from the one significant 
organizational problem that was encountered. According to the director, over the last six years 
there was considerable resistance from a substantial portion of the academic counselors to their 
inclusion or participation in the SAP. This resistance remained a significant problem in the SAP's 
operations at one of the district's high schools throughout the life of the project. This resistance 
apparently stemmed from two important factors. 

First, the more "overt" reason for the resistance was that a "turf battle" persisted between the 
"academic" counselors and the SAP counselors. The SAS/SST structure, described earlier, 
progressively grew more comprehensive in its scope of responsibilities as the program grew. This 
structure was developed to work as a kind of funnel through wliich all students who were not 
performing as expected at school were passed through for assessment. As the scope of the 
S AS/SAP structure broadened, this inevitably infringed upon the academic counselor's 
responsibilities, and some resentment was generated. Part of the reason for moving toward 
integrating academic counseling services into the SAP appeared to be, in part, an effort to resolve 
this issue; if we're all on the same turf, there will be nothing left to fight over. 

According to the director, a second, and probably more powerful, reason for resistance among 
the academic counselors was that some of the counselors were uncertain if they wanted to actually 
involve themselves in their students' emotional and psychological problems. The SAP 
counselors — and the academic counselors if they actively participated in the SAP program — ran 
support groups for students who had a number of problems, including AOD recovery, depression, 
and abuse. The director believed that, for some academic counselors who were not trained to 
perform this kind of service, or for whom this was simply not what they expected in choosing to 
become counselors, this kind of involvement created anxiety. 

In summary, like the other SAPs, this SAP was politically strengthened by its ability to work 
closely with key players and constituencies within the district. Because most district administrators 
and school board members were convinced of its importance to the success of the students, this 
program received strong political support The difficulties encountered with some academic 
counselors, while they were significant, appeared to moderate. In short, this SAP was stable and 
well-integrated into the district's operations. 
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SECTION 3: A REVIEW OF LAST YEAR'S PROCESS 
EVALUATION FINDINGS 

This section provides a description and selected findings from the Process evaluation of the SAP 
demonstration projects conducted during the 1990-91 school year. This review is not an 
exhaustive analysis of the Process evaluation findings. Rather, this section: (a) alerts the reader to 
some of the more important findings; and (b) shows how the Process evaluation played a role in 
shaping the conduct of the 1991-92 Outcome evaluation activities. Readers interested in the full set 
of findings are referred to the original report (Student Assistance Demonstration Project: 
Process Evaluation, Pollard, Horowitz, & Houle, 1991). 

The Purposes of the Process Evaluation 

The Process evaluation was designed to assess the set of activities that occurred within and 
between the service providers and the students participating in the S APs. Descriptions of the 
student subpopulations served, the decisions made regarding the students, and the schedule of 
services provided were all important aspects of this assessment. This Process evaluation 
addressed issues primarily related to program coverage. That is, were students in need of services 
being identified? Also, once identified, were services available to these students? Furthermore, 
the Process evaluation clarified the patterns of services provided in relation to each student's 
presenting problems. 

As mentioned, another goal of the Process evaluation was to guide the efforts for the Outcome 
evaluation of the SAP demonstration projects in the coming year. To accomplish this task, careful 
analysis and descriptions of the monitoring and tracking systems adopted by the individual S APs 
were required. The Process evaluation also addressed issues related to mechanisms for assessing 
the effects of services provided and following up with service recipients. 

The Methodology of the Process Evaluation 

A significant amount of planning was conducted at the start of the project Meetings were held 
with the project site coordinators at the start of the 1990-91 school year to assess the information 
being collected at each site. It was determined, from data samples provided to the evaluation team 
as well as through discussions with site coordinators, that all sites already were collecting 
appropriate descriptive information on students receiving services. 

However, because of organizational and administrative differences among the sites, there were 
significant variations in how the raw data documenting SAP operations were actually collected and 
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transmitted to SWRL. For example, from Mariposa County the data documenting student services 
were taken directly from the project director's personal appointment book, from a ledger that 
documents student services, and from personal inquiries with the director to clarify ambiguities in 
the data or to eliminate as much missing data as possible. This approach was effective in Mariposa 
because the director of the project provided the bulk of the services in the SAP. At the other 
extreme, data from Mendocino County were collected directly from its sophisticated computerized 
data-base system. This system tracked student intakes and services for all activities conducted 
under the umbrella of the MCYP. This computerized system was comprehensive, accurate, and 
relatively easy to access. For the Process evaluation, the data pertaining to those students 
participating in the SAP were simply delivered on disk to SWRL. The relevant information was 
then incorporated into the Process evaluation data-base. 

A comprehensive paper and pencil data collection system was installed in Pajaro Valley prior to 
the Process evaluation. This system provided a detailed tracking system for all students 
participating in SAP activities and was centrally maintained. Based on photocopies of Pajaro 
Valley records, it was a straightforward process to translate the districts data into the Process 
evaluation data base. A similar situation existed for Clovis and San Dieguito. The difference 
between Pajaro Valley and these two sites was that the record-keeping system at Clovis and San 
Dieguito was decentralized. This required that records be retrieved from the different school-sites. 
However, once retrieved, the data were transcribed into the Process evaluation data base. 

Based on the data collected from the SAP sites, a custom data base was constructed by SWRL 
to hold the data collected from the project sites. The data records were maintained at the level of 
the individual student. Information from this data base was used to describe students along a 
variety of demographic dimensions, relate presenting problems to actions taken, to assess the 
availability of services to students, and to determine if other key issues, such as confidentiality, 
were appropriately addressed. Student contacts from Jan. 1, 1991, to June 30, 1991, were 
included in the data base for this Process evaluation. 

Results of the Process Evaluation 

In this section, selected results are reported. (The original format of the Process evaluation report 
is retained here, where a specific finding is detailed along with the implications that were drawn 
from that finding.) The Process evaluation results are organized into two sections. First, analyses 
related to coverage accountability are reported. Coverage accountability represents the extent to 
which the SAPs were effectively reaching the student population they were designed to serve. In 
£ the next section, variables related to service delivery accountability are discussed. This section 

includes the bulk of the analyses provided in the Process evaluation report. 
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Coverage Accountability 



As mentioned, coverage accountability relates to whether the students being served by the SAPs 
were those whom the projects should be serving. In this section, we discuss the numbers of 
students being served by the projects and their demographic characteristics we discussed. 

Table 4 

The Number of Students Served and the Amount of Services Provided to Different 
Student Groups 



Type of students 



Average 
number of 
service events 



Average 
number of 
assessments 



Average number of 
days from ID to first 
assessment 



All students 



6.66 
(JV= 1330) 



1.04 
(N= 1330) 



1.77 
(N = 701) 



All males 



High school males 



Middle-junior high 
males 



5.68 
(N = 482) 

6.27 
(N = 382) 

3.42 
(iV = 100) 



1.07 
(N = 482) 

1.16 

(N = 382) 

.72 
(N= 100) 



1.74 
(N = 257) 

1.80 
(N = 237) 

1.00 
(N = 20) 



All females 



High school 
females 



Middle-junior high 
females 



7.21 
(N = 848) 

7.85 
(W=691) 

4.39 
(/V = 157) 



1.02 
(N = 848) 

1.16 
(/V=691) 

.43 
(N= 157) 



1.79 
(N = 444) 

1.80 
(N = 420) 

1.58 
(N = 224) 



Finding: There were 1,330 students who contributed data to the Process evaluation on 
whom it was possible to obtain a complete set of service related data. Of this 
total, about two thirds (63.8%) of the students were female. 
It appears there were some differences in how services were provided to different 
student groups. It is worth noting that the junior/middle high school students 
received fewer services than the senior high school students. Females appeared 
to receive a somewhat greater number of service events, but otherwise are similar 
to males. 
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Implication: Females were overrepresented in the population of students being served. There 
are several potential reasons why females were overrepresented. This may 
indicate that referrers were more likely to note problems among their female 
students; they may see problems affecting the female student as more appropriate 
for the SAP; or there may, in fact, have been a higher rate of problems among the 
female population. 

Oiie other interpretation appeared more likely, however. Because of the 
"broad brush" approach of these projects, many more services that are of interest 
to females were offered in these SAPs than in S APs of more limited scope. 
Adolescent females may simply be more willing to discuss problems related to 
family and personal relationships more frequently than adolescent males. In fact, 
when SAP services are focused specifically on just AOD services, the usual 
finding has been that males are overrepresented in the service population and that 
females are in the minority. 

These findings suggested that there were some service differences between 
the middle and high school levels. Other findings that follow in the next section 
support this conclusion. A discussion of the implications of these findings is 
postponed until those findings are discussed. 



Figure 1 

Ethnicity of Students Using SAP Services 




Finding: The ethnic breakdown of the students served by the SAPs was similar to the 
ethnicity of the communities (as measured by census data) in which the SAPs 
were located. 



Implication: The projects appeared to fairly and equitably serve all segments of the student 
population as judged by their ethnicity. No particular ethnic group was notably 
over- or under-referred for services. 
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Service Delivery Accountability 



In the next set of analyses, we examined issues related to the conduct of the services provided to 
students in the SAPs. Attention was paid to how and why the students were referred into the SAP, 
how quickly and efficiently services were delivered, and what kinds of services were provided to 
the students within the projects. 

Figure 2 

Sources of Referrals to the SAP 

Other 
Multiple sources 
Off-campus source 
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Finding: More than half of all students referred for SAP services during the 1990-91 

school year (for whom referral source was noted) were self-referred (50.2%). 
Teachers and school counselors were the next major sources of referrals with 
11.3% each. Of the students referred, 472 students (33.4%) had no referral 
source noted. 

Implication: The fact that over half of these students were self-referred may be related to the 
reasons for which referrals are made (more than half of all referrals are for either 
emotional or family problems). Or, the proportion of referrals might also have 
been related to the potential for contact and familiarity with the student, coupled 
with how socially appropriate it was for certain persons to refer in the school 
setting. Self-referrals top the list, followed by teachers and school counselors, 
peers, and other on-campus sources. Furthermore, it appeared that specific staff, 
namely teachers and school counselors, may have had the responsibility of 
referring students to the SAP. 

The high frequency of self-referrals has been noted by the SWRL staff at 
other SAP sites around California, which were not included in the demonstration 
project. If, in fact, self-referral is the primary means whereby students will enter 
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the SAP, this has substantial implications for the design and implementation of 
S APs statewide. For example, a traditional SAP design calls for creating a 
network of school staff who can identify and refer students into the SAP. While 
this is desirable, the S APs may find that facilitating student self-referral is a more 
effective strategy to implementing the SAP. In fact, in an analysis reported in the 
section on students with more serious problems, the students with more serious 
problems had the highest rate of self-referral of any student group. 

It is probably relevant to this issue that S APs started primarily as a way of 
helping students with AOD use. Common to students (and adults) who are AOD 
abusers is a denial that they have a problem. Thus, with chemical dependency, 
self -referral is not very likely; therefore, S APs started out emphasizing the need 
for an effective referral network. With the "broad-brush" model that was been 
adopted in these demonstration projects, the lack of self-referral was not an issue. 

It should be noted that the reasons for referral to the SAP were the reasons 
used by the referral source to indicate why the student was approached (or should 
have been approached by) the SAP for services. These reasons typically reflected 
observable, specific behaviors, indicative of greater emotional distress. These 
referral reasons do not necessarily match the actual underlying issue/causative 
factor(s) for the observed behaviors and symptomatic actions. 



Figure 3 

Reasons for Referrals (Including All Self-Referrals) to the SAP 
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Finding: The most often cited reason for referrals to the SAP was for behavioral problems 
(noted more than 40% of die time). Behavioral problems included antisocial 
behaviors in the classroom, overt depression and/or withdrawal, other overt 
emotional symptoms, physical symptoms, poor attention skills, and criminal 
behavior (e.g., vandalism). Social relations was indicated as the referral reason 
when the student's problem focused primarily on problems in his/her 
relationships with other people, including family members. AOD problems are 
noted approximately 10% of the time, as are school-related issues (including 
attendance, academics, and language problems). There were 442 students for 
whom no referral reason was noted (31.3%). 



Implication: Judging from the results, the SAP projects appeared to help the student with a 
variety of behavioral and social problems that became apparent in the school 
setting. Further, this finding documents that by the second year of the SAPs 
operations, the projects had successfully created "broad-brush" programs, which 
were reaching students with a variety of problems. Thus, these SAPs seemed to 
fulfill the goal of exploring nonacademic solutions to academic problems, which 
is often stated as a goal of SAPs. Despite the opportunity to note multiple reasons 
for the referral, over 75% of the students referred for school-related problems had 
no other problem noted. 
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Figure 4 

Actions Taken by the SAP at First Assessment 



Individual academic svcs 



Referred to CBO 



C/3 

C 

O 
• i— i 
+-» 

o 

< 



School based group 



No r urther action 



School based individual 



1.8% 



W 1% 



13.2% 




38.0% 



38.2% 



i i i | i i i j i i i | i i i | i i i J i i i | i i i | i i i | i i i | i i i | 

0 20 40 60 80 100 120 140 160 180 200 
Number of Referrals 



Finding: The actions taken most often in dealing with student problems were individual 
counseling (school based) and no further action, which together accounted for 
more than 75% of the referrals dealt with in the Process evaluation data. It should 
be noted that no action included continued monitoring by school staff, which 
accounted for 46 (24.6%) of the 187 referrals in this category. Less than 10% of 
the students were referred for services outside the school setting. Of the students 
referred, 922 (65.2%) had no assessment outcome noted. 

Implication: Schools participating in this project used a one-to-one counseling model as the 

intervention of choice in the treatment of most student problems. Only 13.2% of 
the students were referred to groups in the school. This finding was 
disconcerting considering the strong effects noted in the research literature of peer 
programs in helping students with problems. 

There seemed to be a high proportion of students for whom no further 
action was recommended by the SAP. Less than one fourth of these students 
were expected to receive continued monitoring by school staff. This high rate of 
inaction may have been due to the lack of options for dealing with students. 

The low rates of use of community-based organizations could indicate a lack 
of potential referral options (a serious problem in a site like Mariposa) or may 
positively reflect the projects 1 ability to handle their students 1 problems in-house. 
In the data collected related to students who were assessed a second time, the 
proportion referred to services outside the school setting increased to 23.4% at the 
second assessment This indicated that schools prefer to attempt to deal with 
students internally first and use outside resources as a second option. However, 
a second or third reassessment was probably associated with students who were 
experiencing more severe difficult^ s, and referral to outside professional services 
might have been expected for that reason.. 
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Students With Serious Problems 



A student was classified as having a serious or major problem if any of the following were noted in 
the presenting problems list: parental AOD use, sibling AOD use, family divorce/breakup, sexual 
abuse, physical abuse, severe depression, grief or loss, pregnancy, personal trauma, delinquency, 
and/or gang activity. Because of the importance of quickly identifying and serving students who 
are in serious distress, special analyses looked at some of the service delivery features associated 
with these students. 



Figure 5 

Proportion of All Referrals With Serious Problems 




All sites High schools Middle/junior highs 

Sites 

Finding: Approximately 35% of all referred students were identified as having major 
problems. Forty percent of the high school students and 14.7% of the 
middle/junior high students were classified as having major problems. 



Implication: The proportion of the referrals of students with major problems to SAPs was 

much higher in the high schools than middle/junior high schools. These students 
were recorded as more likely to be dealing with a number of the issues in the 
major problem list. In addition, by the time these students reach high school the 
problem was more likely to be salient. That is, it was more likely to be of higher 
severity, more likely to be acted cut in the school setting, and more likely to be of 
significance to the students themselves. 
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Figure 6 

Referral Source 



70-, 





Major problem 
ffl No major problem 



Self 



Family Peer Teacher School Other on- Off- Multiple Other 

counselor campus campus 
Referral Source 



Finding: Approximately 64% of students with major problems self-referred compared to 
35.6% of students without major problems. 

Implication: Students with major problems were almost twice as likely to self-refer as students 



without major problems. This was probably due to several reasons. First, the 
greater distress caused by severe problems prompted the student to seek services. 
Also, the major events associated with this type of student (e.g., death, family 
violence) were more concrete and recognizable to students as a reason for seeking 
out services. Because these students were probably more aware of their problem, 
and if they sought out services promptly, there was not much time for the 
students' problems to be noticed by others at school. Thus, they tended to be 
self-referred rather than referred through someone else. Students with major 
problems also were three times more likely to be referred from off-campus 
sources (typically, law enforcement) as students without major problems, 
although referrals from off- campus sources were infrequent. This seems 
appropriate considering the types of problems for which students from off- 
campus sources were typically referred (Figure 6). 
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Table 5 

Amount of Services Delivered to Students With Serious Problems 



Type of students 



All students 



All males 



Males, not serious 
problem 

Males, serious 
problem 



All females 



Females, not 
serious problem 

Females, serious 
problem 



Average number of 
service events 



6.65 
(N = 1358) 

5.58 
(N = 492) 

3.68 
(N = 341) 

9.87 
(N= 151) 

7.25 
(N = 866) 

5.30 
(N = 318) 

10.62 
(N = 548) 



Average 
number of 
assessments 



1.05 
(N = 1358) 

1.08 
(N = 492) 

.84 
(N = 341) 

1.64 
(N = 151) 

1.04 
(N = 866) 

.71 
(N= 318) 

1.61 
(N = 548) 



Average number of 
days from ID to first 
assessment 



1.76 
(N=717) 

1.74 
(N = 262) 

2.19 
(N = 135) 

1.26 
(N = 127) 

1.77 
(N = 455) 

2.26 
(N = 196) 

1.41 
(N = 259) 



Finding: It appeared that the service delivery times, in general, were quite prompt for all 

students. Students appeared to receive services quickly after they were identified. 
More importantly, students who were identified as having more serious problems 
received more services, received more assessments, and received quicker 
assessments than students who were recorded as having less serious problems. 

Implication: The implication of these findings is straightforward. By their second year of 

operation, the SAPs were doing a credible job at identifying those students who 
were reporting more serious presenting problems, and the SAPs appeared to be 
adjusting their service delivery patterns appropriately when such a student was 
encountered. In general, they appeared to be delivering services to students about 
as quickly as could be expected. 
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Conclusions to il:e 1990-91 Process Evaluation 



In the Process evaluation analyses, SWRL undertook a detailed look at the demographic 
characteristics of students being served, the kinds of problems the students brought to the SAPs, 
the referral system that funneled the students into the system, and how the SAPs served the 
students once they were identified. The following summarizes the key findings and conclusions: 

♦ By the second year of the SAPs' operations, students were being served by the 
SAPs in rough proportion to their sex and ethnicity in the overall student 
population. There did not appear to be any student subpopulation that was 
significantly under- or over-served by the SAPs as they were conducted in the 
1990-91 school year. Female students, in general, received more of the services. 
On average, about 10% of each of the school's students were being served by the 
SAPs during the Process evaluation. 

♦ By far, the most important source of referrals for the SAPs was from 
self-referrals. This source of referral increased in importance when 
consideration of the seriousness of the student's presenting problem was made; 
self -referrals made up an even higher percentage of the referrals with students who 
reported more serious presenting problems, such as family violence, personal 
trauma (e.g., rape), family divorce or breakup, or delinquency. Teachers, school 
counselors, and other on-campus sources made up the next three most important 
referral sources. 

♦ The implication of the above conclusion is that the SAPs did a very good job 
establishing a high level of credibility and trust with the students in a short 
time period. Without this level of credibility and trust it is highly unlikely that the 
students would have disclosed such serious and emotionally painful experiences 
and feelings. 

♦ About one third of the students were recorded as having a serious 
personal problem when they entered the SAP. The project directors reported 
to SWRL staff that large numbers of students were bringing very serious problems 
to the SAP. The data recorded by the Process evaluation supported this 
observation. 
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• On average, the students received about 6.6 service events. The average 
wait from the identification of the student to his/her first assessment was less than 
two days. There were no significant differences in these figures between males and 
females or between middle and senk ~ high school students. However, as would be 
hoped, there were significant differences between the students who were judged to 
have serious presenting problems and the remaining students. For the more 
serious students, there was a substantial increase in the number of 
service events, the number of assessments increased, and there was a 
shorter time lag between identification and the first assessment event. 

The two most frequent reasons for referring the students to the SAP 
were (a) behavioral problems and (b) social relationship problems. 

Behavior problems accounted for about 40% of the referrals, and included 
antisocial behaviors, students who were seen as overtly depressed and/or 
withdrawn, other overt emotional symptoms, physical symptoms, poor attention 
skills, and criminal behavior. AOD problems were noted about 10% of the time. 
Students in high school were more likely to have multiple problems noted in their 
records. 

The most frequently noted actions taken at assessment were (a) referral 
of the student to school-based individual services or (b) "No Further 
Action." The "No Further Action" category included a specific category for 
continued monitoring of the student by the SAP staff even though no specific 
service was immediately going to be provided. The "No Further Action" category 
dropped in frequency among students who were noted to have serious problems. 

Overall, it appears that by the second year of their funding, the S APs were functioning as 
planned. They were serving all segments of their student populations; they were successfully 
identifying a large proportion of students who were reporting serious problems; and they were 
serving all students efficiently and frequently. Few, if any, organizational problems could be 
identified in the Process evaluation. The only concern that was identified at this point in the 
evaluation was that some of the project sites could improve their record-keeping systems. It was 
anticipated that with the demands placed upon them by the upcoming Outcome evaluation in this 
area, a considerable amount of the Outcome evaluation effort would be devoted to improving these 
systems. 
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SECTION 4: THE OUTCOME EVALUATION 

This section of the report details all activities that occurred as part of the 1991-92 school year's 
Outcome evaluation effort The third year of the evaluation focused on measuring the effectiveness 
of the demonstration S APs. It is by far the most important year of the evaluation. Prior to 
measuring program outcome, it is important to confirm that the services were being delivered as 
intended. In other words, unless it's certain the project was delivering services as designed, it's 
unfair to test whether the project was successful. 

The Delivery of Program Services 

The activities documenting program implementation replicate a subset of the analyses conducted as 
part of the 1990-91 school year's Process evaluation. The primary question for the Outcome 
evaluation is whether program services were delivered as intended to the participating students. 
Also, did the S APs provide enough services to make it plausible that the program could produce 
beneficial effects? Other Process evaluation issues, discussed in detail last year, were not crucial to 
this phase of the evaluation. For example, there was no reason to believe that referral patterns 
would change significantly from year to year. As a result, an analysis of referral sources or 
examining the relationships between referral sources and presenting problems was not the focus of 
this year's evaluation effort. 

Student Demographics 

Table 6 shows the number of males and females from both the middle school and high school 
levels participating at each SAP site. There was a preponderance of females participating in the 
evaluation, with about two thirds of the students being female. This was particularly true for 
Mariposa at the high school level, where more than four out of five students were female. 

The student demographics were based on all students who entered the SAP during the pre-test 
phase of the project. Note that about 1 1% of the students did not have their sex recorded in the 
file, resulting in missing data. 

Delivery of Services 

Table 7 summarizes the delivery of services for those students participating in evaluation through 
the post-test phase. The information in Table 4.2 is broken down by whetner the student entered 
the SAP with what was classified as a serious presenting problem. A student was classified as 
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Table 6 

The Numbers of Male and Fen ile Students Participating in the SAPs at the High School and 
Secondary School Levels by Each District 





Female 


Male 


N 


All Students 


.66 


.34 


905 


High School 


.72 


.28 


559 


Clovis 


.73 


.27 


114 


Mariposa 


.81 


.19 


53 


Mendocino Co. 


.79 


.21 


178 




.66 


.34 


97 


San Dieguito 


.60 


.40 


117 


Middle/Junior High 


.57 


.43 


346 


Clovis 


.54 


.46 


149 


Mariposa 


.39 


.61 


38 


Pajaro Valley 


.71 


.29 


94 


San Dieguito 


.54 


.46 


65 



Note. Total cases = 1020. 

Note. Missing cases = 115 or 11.3%. 

Note. Mendocino County had no middle schools in the evaluation. 

having a serious presenting problem if any of the following were noted in the presenting problems 
list: parental AOD use, sibling AOD use, family divorce/breakup, sexual abuse, physical abuse, 
severe depression, grief-related problems, pregnancy, personal trauma, delinquency, and/or gang 
activity. Because of the importance of quickly identifying and serving students who are in serious 
distress, the evaluation has consistently looked at some of the service delivery features associated 
with these students. 
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Table 7 

The Average Number of Days from ID to First Service, and the Average Number of Total 
Service Events, Individual Service Events, and Group Service Events 



Service events 





Days 


Total 


Individual 


Grojp 


N 


For All Students 


2.85 


11.35 


2.75 


7.52 


820 


Not High Risk 


4.42 


10.24 


1.98 


7.24 


495 


Clovis 


1.00 


13.21 


2.71 


6.87 


91 


Mariposa 


1.00 


12.87 


2.67 


9.58 


55 


Mendocino Co. 


1.04 


22.06 


2.55 


19.51 


67 


Pajaro Valley 


XT/ A 

JN/A 


0.4o 


1 Q < 


O.ZZ 


1 A~> 


San Dieguito 


12.45 


3.43 


0.97 


1.72 


140 


Students With Serious 
Presenting Problems 


1.29 


13.05 


3.92 


7.96 


325 


Clovis 


2.57 


14.88 


5.03 


8.43 


137 


Mariposa 


1.00 


14.36 


2.53 


11.28 


47 


Mendocino Co. 


1.00 


24.10 


2.62 


19.86 


21 


Pajaro Valley 


N/A 


7.25 


2.72 


4.14 


59 


San Dieguito 


2.00 


9.70 


4.11 


3.93 


61 



Note. N/A = Not available. 
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The number of students on whom complete information regarding program participation could 
be obtained was 820 (see Table 7). This means that 200, or approximately 20% of the students, 
were lost from the study prior to post-test data collection. Many of these students were lost 
because they left the district, transferred to another school, or in some other way los* contact with 
the SAP program at the site during the Outcome evaluation period Other students were lost from 
the study because the SAP sites failed to appropriately collect follow-up data. It is impossible to 
determine how many of the students were iost to each reason. In addition, one of the four school 
districts participating in Mendocino County dropped out of. the evaluation. This loss represented a 
small but significant number of students. 

Table 7 presents information on the number of days from identification (ID) to first service and 
on the total number of service events, as well as the number of individual counseling and group 
counseling service events. While the general service delivery pattern is similar to last year's 
findings, some changes are worth noting. 

First, as found last year, the SAP sites provided one third more services to students with 
serious presenting problems than to other students. 

This year the SAP sites provided more services to students. This is probably due to more 
efficient functioning of the programs with increased maturity, combined with improved record 
keeping. (Participating in the evaluation had the effect of improving record keeping.) The most 
active program, Mendocino County, also was the program that functioned, with the smallest 
budget. The heavy reliance on groups is evident in Table 7, especially with Mendocino County, 
Group services were by far the most common service modality. 

Compared to last year, there was a slightly longer time between the initial identification/referral 
of a student and his/her first service event. The average time was skewed higher by the inclusion 
of the San Dieguito data While San Dieguito might quickly send the student through the core team 
process, this site apparendy was much slower than the other sites to begin providing services. 
However, San Dieguito began services fairly prompdy for the students with serious presenting 
problems. It is a reasonable assumption that the overall increase in the response time for the S APs 
as a group is due to the larger numbers of students and services being provided in the 1991-92 
school year. Because of the increasing service load, it probably was not possible to provide the 
immediate response that was evident in last year's Process evaluation report. Nevertheless, many 
of the sites initiated services within a day or two, which is very prompt compared to any other 
social service delivery system. In fact, the speed of service delivery is one of the most positive 
features of the S APs. For example, a referral for a non-life threatening condition to county mental 
health services often results in an appointment in two to three weeks. In comparison, the S APs 
look very good. 

Based on the data in Table 7, we conclude that the SAP demonstration sites generally provided 
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services as intended. Given that the services were being delivered, we now turn to the 
effectiveness of the services. 

SAP Impact on Student Alcohol, Tobacco, and Other Drug Use 
The California Student Survey 

The measure of student AOD use was the California Student Substance Use Survey (CSSUS, 
Skager & Austin, 1993). This instrument is used in the California Attorney GeneraTs biennial 
survey of student AOD use. The CSSUS, because of its repeated use in California school settings 
over the past eight years, has well-documented California norms for student AOD use. The 
CSSUS is available in both Spanish and English versions and has a reading level of approximately 
sixth grade. 

The CSSUS measures the six-month frequency of student use for a wide variety of substances: 
beer, wine, hard liquor, marijuana, hashish, methamphetamine, cocaine, inhalants, cigarettes and 
snuff. In addition, SWRL included two CSSUS questions assessing the student's appraisal of the 
harmfulness of regular marijuana and alcohol use. For each of the items, students respond on a 
eight-point scale, ranging from not having used the substance at all in the past six months to having 
used the substance every day or nearly every day. 

The Effects of the SAPs Upon Student AOD Use 

Table 8 shows the means and number of students on whom CSSUS data was collected, both pre- 
and post-test. Data in this table are reported for the combined SAP sites versus the comparison 
site. 

More students completed pre-test than post-test data. As was discussed, this is due to a 
combination of routine student attrition, the loss of one school district in Mendocino County, and a 
failure of 100% follow-up in two of the districts. Because the evaluation staff could rely on 
centrally maintained records for measuring service delivery, we had somewhat less student attrition 
when collecting that data. The need to physically recall the student for the post-test data collection 
on the CSSUS (and the other instruments) resulted in a small, further loss of students from the 
study. Overall, the SAP sites collected post-test data on 72% of the students on whom pre-test 
data was collected. The loss of about 28% of the students is common in evaluations conducted in 
multiple school-sites across multiple districts and where there is a substantial time lag from pre- to 
post-testing. 
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Table 8 

Student Six-month Frequency of Use Rates on All CSSUS Items 



Pre-test 



Post-test 







n 


Mean 


n 


Mean 


Beer 


S APs combined 


1,010 


2.56 


729 


2.59 




Comparison site 


1,042 


2.16 


1,498 


2.67 


Wine 


SAPs combined 


1,008 


2.22 


729 


2.19 




Comparison site 


1,035 


1.92 


1,508 


2.38 


Liquor 


SAPs combined 


1,007 


2.03 


727 


2.02 




Comparison site 


1,037 


1.78 


1,511 


2.25 


Marijuana 


SAPs combined 


1,006 


1.93 


726 


1.96 




Comparison site 


1,031 


1.44 


1,511 


1.96 


Hashish 


SAPs combined 


1,001 


1.15 


727 


1.16 




Comnarison ^ite 


1,032 


1.1 1 


1,511 


1.34 


Methamphetamine 


SAPs combined 


1,001 


1.13 


729 


1.14 




Comparison site 


1,030 


1.09 


1,509 


1.34 


Cocaine 


SAPs combined 


1,003 


1.13 


729 


1.13 




Comparison site 


1,031 


1.10 


1,513 


1.30 



table continues 



40 



51 



Pre-test 



Post-test 



Inhalants 



Cigarettes 



Snuff 



Marijuana 
harmfulness 



Alcohol 
harmfulness 



S APs combined 
Comparison site 

S APs combined 
Comparison site 

SAPs combined 
Comparison site 

SAPs combined 
Comparison site 

SAPs combined 
Comparison site 



n Mean 

1,000 1.20 

1,030 1.21 

1,009 1.80 

1.038 1.37 

1,009 1.13 

1.036 1.20 

978 1.84 

1.039 1.77 

978 1.77 

1.037 1.94 



n Mean 

728 1.22 

1513 1.45 

728 1.78 

1503 1.60 

727 L13 

1504 1.34 

692 1.84 

1493 1.86 

692 1.71 

1487 1.96 



It also is worth noting that SWRL obtained a larger number of students at the comparison site 
during post-test than during pre-test. This was partially due to very rapid growth of the 
comparison site district. More importantly, by the post-test period, SWRL garnered greater 
cooperation from the individual teachers at the comparison schools, thus generating greater student 
participation. 

The results on Table 8 indicate that the SAP sites appeared to suppress AOD use in 
participating students. Looking at the comparison site values first, note that for all substances, the 
six-month prevalence figures increased. In other words, students at the comparison schools 
reported increased use of all substances across time. This finding of increased use over time is 
expected; student use of all substances is known to increase over time throughout adolescence. It 
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is not until around age 19-20 that substance abuse levels off. 

In contrast, the findings for the SAP sites showed that, for all substances, use levels remained 
stable from pre-to post-test. In other words, students from the SAP sites did not increase their 
substance use. It appears that the SAP sites effectively halted the increasing levels of use that 
would otherwise be expected in an adolescent population. 

Statistical analyses confirm this interpretation. Multiple Analysis of Variance (MANOVA) was 
used to test the hypothesis that the SAP sites suppressed AOD use. MANOVA is a statistical 
procedure contrasting the SAP sites versus the comparison sites for differences across multiple 
AOD substances (i.e., multiple dependen. variables). Conducting a separate analysis for each of 
the AOD substances would lead to an overestimate of experimental effects because of the increased 
likelihood of taking advantage of chance effects in the data. MANOVA corrects this problem, 
allowing for an accurate assessment of program effects. 

The key analysis from the MANOVA is a test of the interaction between the experimental group 
(i.e., SAP vs. comparison site) and the measurement occasion (i.e., pre-test vs. post-test). This 
interaction tests the basic question of whether the two experimental groups had different rates of 
increase in AOD use. In other words, the MANOVA analysis can determine whether AOD use is 
increasing at a significantly faster rate for the comparison group than for the combined SAP sites. 
The results (see Table 9) showed that, in fact, the AOD use for the comparison group was 
increasing at a significantly faster rate than the SAP sites (F = 3.52, Hotelling's T = .012, 
p <. 001). These results confirm the interpretation that the students participating in the SAPs were 
showing no increase in substance use, while the comparison sites were increasing use at a 
significantly higher rate. 

Given that the overall F-test was significant, it was appropriate to examine the findings for the 
individual AOD substances. For all substances, the SAP sites had a significantly lower increase in 
use as compared to the comparison sites. The effect was strongest for beer, wine, liquor, and 
marijuana, probably for two reasons. First, statistically it was easier to detect an effect with these 
substances because of the initially higher levels of use because more students use these substances. 
Second, students using the other illicit substances are probably more difficult cases and would be 
less likely to show program effects, given the amount of SAP services being delivered. 

While significant, the effect of the SAPs on suppressing cigarette and snuff use was more 
limited than for other substances. Also, while the SAP students showed an increasing awareness 
of the harmfulness of marijuana, there was no observable effect for alcohol. 

For the AOD data, the difference between the SAP and comparison sites translates into an 
effect size of approximately .2. An effect size is a standardized metric that allows for comparison 
of the magnitude of program effects across different studies. An effect size of .2 is generally 
viewed as a modest research effect (Cohen, 1977). 
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Table 9 

MANOVA Results Comparing Pre- to Post-test Changes in ATOD Use and Assessment of 
Alcohol and Marijuana Harmfulness for the SAP Sites Versus the Comparison Site 





FTest 


Sig. 


All substances 


3.53 


< .001 


Beer 


14.60 


< .001 


Wine 


22.45 


< .001 


Liquor 


17.60 


< .001 


Marijuana 


14.78 


< .001 


Hashish 


10.91 


.001 


Methamphetamine 


11.32 


.001 




8 34 


.004 


Inhalants 


6.07 


.014 


Cigarettes 


6.59 


.010 


Snuff 


8.04 


.005 


Marijuana harmfulness 


8.05 


.005 


Alcohol harmfulness 


1.89 


n.s. 
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More useful than the effect size to policymakers is a concrete representation of the proportion 
of the SAP students who attained a given level of benefit as compared to the comparison group. 
One way to conceptualize this is by using the Binary Effect Size Differential (BESD). The BESD, 
in the context of this evaluation, is a measure of the proportion of the SAP group, in comparison to 
the comparison group, that did better than the average student in AOD use. An effect size of .2, 
through algebraic calculations, converts to a BESD of 55% for the SAP students and 45% for the 
comparison students. This means that 55% of the SAP students did better than the average student 
in avoiding AOD use in the past six months, while only 45% of the students in the comparison 
group did better than the average student in AOD use. The 10% differential is essentially the 
impact of SAP participation. Given the seriousness of AOD use, this 10% differential implies a 
significant, policy relevant impact. 



Differences Among SAP Sites 



There were substantial differences among the SAP sites in terms of the overall level of AOD use 
(F = 6.31, p < .001). This indicates the obvious: Some of the SAP sites had higher prevalence 
levels in AOD use than others. For example, Mariposa, Mendocino County, and San Dieguito all 
had the highest levels of beer consumption during pre-test and at post-test It is important to 
remember that it is not the absolute level of AOD use that characterizes the effectiveness of the 
S APs; instead, it is whether the level of use moves in an upward and downward trend relative to 
the SAP sites. As reported above, the SAP sites as a whole showed a general downward trend in 
AOD use relative to the comparison site. However, no differences in effectiveness among the 
individual SAP sites were observed (F < 1.0, nonsignificant [n.s.]) for AOD use; all of the SAP 
sites were about equally effective in suppressing AOD use among their participating students. 

SAP Impact on the Social and Psychological Functioning 

Of the Students 

The Piers-Harris Children's Self-Concept Scale 

The Piers-Harris Children's Self-Concept Scale (PH) is a widely used, standardized psychometric 
instrument that assesses the psychological and social functioning of adolescents. While titled a 
self-concept inventory, the PH assesses adolescent social and psychological functioning across a 
variety of domains. The scoring system for the PH produces a Total score and scores for several 
subscales: Behavior, Intellectual & School Status, Appearance and Physical Attribuves, Anxiety, 
Popularity, and Happiness & Satisfaction. Because the survey had never been translated into 
Spanish, SWRL, with the approval of the publisher, translated and reproduced the instrument in 
Spanish. 

The Effects of the SAPs Upon Student Social and Psychological Characteristics 

Table 10 shows the pre- and post-test PH values (Total, all PH subscales, as well as a measure of 
inconsistency in student responses) for the combined SAP sites and the comparison site. As with 
the AOD scores, post-test data was collected on about 73% of the students participating in the pre- 
test data collection phase. In contrast to the AOD prevalence data, the pre-test results showed that 
the comparison site had an initial advantage over the SAP sites. (Higher values indicate better PH 
scores.) Because of the expense of the Piers-Harris, every third student was randomly selected at 
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the comparison site to complete the Piers-Harris. This resulted in a lower number of comparison- 
site students providing PH data as compared to the CSSUwS data. However, this procedure still 
provided a sufficient number of students for adequate statistical power, and the random distribution 
of the PH to the comparison site students ensured a nonbiased sample. 

Table 10 shows that, from pre- to post-test, the combined SAP sites improved on the PH 
relative to the performance of the comparison site. This same pattern was found with the AOD 
data. Compared to pre-test values, the SAP improved on all of the PH subscales. This, of course, 
is reflected in the SAP total score. In contrast to the SAP sites, students from the comparison site 
showed an improvement in two subscales (Appearance & Attributes and Popularity), while there 
was a decrease in the remaining subscales. On the Total score, the comparison site students 
showed a nonsignificant decrease. 

MANOVA again was the procedure used to statistically assess the SAP versus comparison site 
differences. Table 1 1 presents these data. Overall, the assessment of the experimental group by 
measurement occasion interaction showed a significant result (F = 3.60, Hotelling's T = .010, 
p < .001). In other words, the SAP students significantly improved on the PH from pre-test to 
post-test while the comparison site students did not show improvement. In addition to the overall 
F value, the univariate analyses of the PH subscales showed a significant advantage for the SAP 
sites on the Behavior, Intellectual & School Status, Anxiety, and Happiness subscales. In short, 
the Piers-Harris findings support the conclusion that the S APs were producing positive 
psychological and social changes in their students. 

As with the AOD data, the difference between the SAP and comparison sites translates into an 
effect size of approximately ,2. As noted earlier, this is generally viewed as a modest research 
effect (Cohen, 1977). Again, one of the more effective ways to understand this size of an effect is 
to calculate the BESD. An effect size of .2 translates into a 10% differential favoring the SAP 
sites. In other words, while 55% of the SAP students did better than the average student on the 
PH, only 45% of the students from the comparison group did better than the average student. 
Given that the underlying social-psychological characteristics of students are difficult to influence, 
a 10% differential following participation in the SAP suggests that the SAPs were effectively 
helping the students. 

As with the AOD data, there were no important differences among the SAP sites on the PH. 
All appeared to have a similar effect on improving the students 1 PH scores from pre- to post-test 
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Table 10 

Student Responses on the Piers-Harris Total Score, Subscales, and Inconsistencies 







Pre-test 


Post-test 






n 


Mean 


n 


Mean 


Total Score 


SAPs combined 


1,014 


50.23 


731 


52.64 




Comnarison site 


596 


54.48 


307 


53.10 


Behavior 


SAPs combined 


1,014 


10.97 


737 


11.29 




Comnarison site 


596 


12.11 


307 


11.50 


Intellectual & 


SAPs combined 


1,014 


10.53 


737 


11.11 


school status 














Comparison site 


596 


11.46 


307 


11.33 


Appearance & 


SAPs combined 


1,014 


7.73 


737 


8.28 


attributes 














Comparison site 


596 


7.91 


307 


8.04 


Anxiety 


SAPs combined 


1,014 


7.89 


737 


8.78 




Comnarison site 


596 


9.37 


307 


9.04 


Popularity 


SAPs combined 


1,014 


7.87 


737 


8.09 




Comparison site 


596 


8.09 


307 


8.17 


Happiness & 


SAPs combined 


1,014 


6.96 


737 


7.39 


satisfaction 














Comparison site 


596 


7.60 


307 


7.29 


Inconsistency 


SAPs combined 


1,020 


1.98 


746 


1.92 




Comparison site 


597 


1.49 


307 


1.80 



Table 11 

MANOVA Results Comparing Pre- to Post-test Changes in the Piers-Harris Subscales for the 
SAP Sites Versus the Comparison Site 



F Test Sig. 



All Scales 


3.60 


< 


.001 


Behavior 


7.58 


< 


.006 


Intellectual & school status 


5.60 


< 


.018 


Appearance & attributes 


1.42 




n.s. 


Anxiety 


13.57 


< 


.001 


Popularity 


< 1.00 




n.s. 


Happiness & satisfaction 


11.25 




.001 



Student Satisfaction With SAP Services 

Overall Findings 

The students participating in the SAPs were asked a number of questions to gauge their satisfaction 
with the progran. Twenty items measured satisfaction with the overall program, while another 17 
items asked specific questions about the students' group experience. Table 12 presents selected 
survey items assessing the overall program and Table 13 presents selected items assessing the 
group experience. In general, the students appear satisfied with all aspects of the SAPs' 
functioning. 

For items measuring overall program characteristics, students responded using the following 
five point scale: 1) Strongly Disagree, 2) Disagree Somewhat, 3) Neutral, 4) Agree Somewhat, 
and 5) Strongly Agree. The students reported positive judgments regarding staff and student 
confidentiality, the caring and knowledge of the staff, whether the program was helpful, and 
whether they were glad they participated in the program (see Table 12). Across all of the SAPs, 
the average score always was greater than 4.0. 
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The items on Table 13 related to group characteristics. Because of the prominence of group 
services, these items were especially important Students responded to these items using a three 
point scale: 1) Never True, 2) Sometimes True, and 3) Always True. As can be seen for the 
group items, the students were again uniformly positive regarding their involvement with student 
groups. 

Differences Among SAP Sites 

A one-way analysis of variance was conducted for each survey item, and for the average score 
across all items, to determine whether there were differences among the SAP sites. Because of the 
number of items, and to avoid taking advantage of chance, only those items with significance less 
than p < .002 were noted in Table 12 as being significant 

The first item on Table 12 is the average score of all survey items related to overall program 
characteristics. As is evident, there are some differences among the SAP sites. Analysis of 
variance showed that a significant difference (F = 7.53, p < .0001) existed among the sites. 
Mendocino County was the site most favored by students, while San Dieguito was the site least 
favored by students. 

The first item on Table 13 represents the average value across all items relating to the students' 
group experience. As opposed to the overall program items, there were no significant differences 
among the SAP sites; the students rated all of them approximately equal. Also, there were few 
significant differences among the sites on the individual items (see Table 13). 

Summary of the Findings 

The SAP sites performed well in contrast to the comparison sites in all of the analyses. Initial 
review of the amount of service data revealed that the SAP sites were actually providing more 
services that was documented last year. And, as with last year, they were providing more services 
to students with serious presenting problems. The SAP sites also demonstrated a suppression of 
AOD use among their participating students and an improvement in psychological and social 
functioning as measured by the Piers-Harris. Finally, the students participating in the S APs 
reported that they were generally happy and satisfied with services. In short, all of the evidence 
presented suggests that S APs do help students cope with a broad range of problems. 

Three other considerations must be kept in mind. Two of these considerations suggest that the 
S APs may be even more effective than the current findings document. First, it should be 
remembered that the SAP students were compared against the student body as a whole at the 
comparison site. Originally, it was planned that teachers at the comparison schools would nominate 
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students who would benefit from participation in a SAP if one existed at that site. This process 
would identify a subgroup of students who would be somewhat better matched with the students 
participating in the SAPs. However, it was impossible to maintain teacher participation at the 
comparison schools throughout the evaluation process. There simply was no incentive for the 
teachers to actively support the evaluation. As a result, the student body as a whole at the 
comparison site became the comparison group. 

Because of this, the comparison group was most likely made up of students who were, on 
average, less likely to have significant personal problems; these students also were somewhat less 
likely to report AOD use or poor Piers-Harris scores. The improvement of the SAP students 
relative to the comparison site students, even when the SAP students should be expected to report 
more personal difficulties, tends to strengthen the confidence that can be placed in the positive 
effects of the SAPs. 

A second consideration, also supporting the positive results, is that the primary outcome 
measures consisted of AOD use and the Piers-Harris inventory. For most students, AOD use was 
not the primary reason they entered the SAP. Also, for many students, one or more of the Piers- 
Harris subscales also were not relevant to their problems. However, a standard battery of 
instruments that effectively captured all potential student problems could not be provided. Because 
AOD use and self-esteem are known to be related to life difficulties in adolescence, these measures 
provided the best, albeit indirect, measures of program success. However, because the AOD scale 
and the Piers-Harris were indirect measures of program outcomes for many students, this would 
also tend to decrease the likelihood of detecting positive program effects. The fact that these 
measures still demonstrated positive effects strengthens the conclusion that the SAPs were doing a 
good job providing services. 

In contrast to these two considerations, one problem in the evaluation design tends to weaken 
the conclusions regarding positive program effects. Because students were likely to enter the SAP 
when their problems were particularly bothersome, many students would improve over time 
without any particular intervention. This effect is called "regression to the mean." This technical 
problem is an issue in this evaluation because the comparison group students were not identified 
and selected in exactly the same way as the SAP group. The regression to the mean phenomenon 
suggests that the SAP students would tend to perform better than the comparison site students even 
if they never received SAP services. However, for the AOD measures, regression to the mean— 
i.e., returning automatically over time to a more normal pattern of behavior — would mean an 
increase in AOD use because an increase in AOD use during adolescence is the normal pattern of 
behavior. The SAP students did not exhibit this pattern. 

In short, the potential technical complications in the evaluation, on balance, seem to strengthen 
the confidence in the evaluation's positive findings. While the regression to the mean phenomena 



could artificially make the S APs look good, the other technical complications worked against the 
discovery of positive program effects. This situation is common in program evaluation research, 
where most evaluation problems work to conceal positive program effects (Lipsey et al., 1985). 
All things considered, SWRL believes the SAP demonstration projects did successfully intervene 
in the lives of the students they served and that the positive program outcomes accurately reflect 
program operations. 
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SECTION 5: SUMMARY AND RECOMMENDATIONS 

This section addresses two key evaluation questions: (a) Were the demonstration S APs effective in 
addressing a broad range of student problems?; and (b) can a standard SAP program model be 
identified to assist in the dissemination and implementation of S APs in California? In addition, this 
section concludes with several recommendations that S WRL believes will foster the development 
and implementation of S APs in California, These recommendations are equally applicable to the 
national SAP movement. 

Did the SAPs Help Students? 

Findings in the previous section suggest that demonstration SAPs did help students deal with a 
wide range of problems. First, the SAPs delivered services to students who were referred to the 
program for a variety of reasons. Service delivery was particularly efficient for students presenting 
serious problems. Second, the SAPs served a relatively large number of students at each site. 

These results suggest modest, but important, positive effects for students participating in the 
SAPs. The results indicate two effects from SAP participation. First, students in the SAPs 
reported no increase in the AOD use, while the comparison group students showed an increase in 
AOD use. This increased substance use over time is a standard finding in adolescent populations. 
Helping students not increase use over time indicates an effective intervention. Second, the SAP 
students demonstrated improvements in social and psychological functioning as measured by the 
Piers-Harris Children's Self-Concept Scale, while the comparison site students showed no 
improvements. 

Students generally reported high levels of satisfaction with the SAP services. Students also 
recognized important program features, such as student and staff confidentiality. They felt the 
services helped them with their personal problems. Students reported that group services provided 
positive benefits, were an emotionally safe place to explore their feelings, and they developed a 
strong commitment to their participation in groups. 

While there were technical limitations in the experimental design, these limitations tended to 
work against the discovery of positive effects. In two cases, the limitations would tend to make 
the SAP students look worse in contrast to the comparison students, while one limitation could 
work to make the SAP students look artificially good. This situation is common in program 
evaluation research, where most evaluation problems work to conceal positive program effects 
(Lipsey et al., 1985). However, on balance, the potential technical complications in the evaluation 
seem to strengthen the confidence in the evaluation's positive findings. All things considered, 
SWRL believes that the SAP demonstration projects successfully intervened in the lives of the 
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students they served and that the positive program outcomes accurately reflect program operations. 



Can a Standard SAP Program Model Be Developed? 

The second major question posed for this evaluation was whether a standard SAP program model 
could be developed to facilitate the dissemination of SAPs throughout California. We weren't able 
to document the superiority of one particular SAP model, as encountered in the demonstration 
sites, over another. It is not clear from this evaluation that there is any organizational structure that 
best facilitates the delivery of services. Particularly with regard to the measured program 
outcomes, there does not appear to be any performance advantage by any of the SAP sites. 

However, this conclusion of equivalence across the SAP sites must be tempered by one 
consideration. The students were more satisfied with services at some SAP sites as compared to 
others. The two districts that had the oldest, and most formal, organizational structures for their 
SAPs fared the worst in student satisfaction. But, it is not clear whether it is the formality of the 
organizational structure, or the age of the program, or even some other factor, that accounts for 
these findings. Larger SAP organizations might be intimidating to students and their families. A 
larger, more formal organization would work against the informality and ease of access that are a 
hallmark of SAPs. However, the higher levels of student satisfaction at the other three sites might 
just as easily be due to the newness of the programs. When new programs, with new personnel, 
are implemented, there is a high level of enthusiasm and support for the program that is typically 
not maintained over several years. The decline in enthusiasm is inevitable and could easily be the 
cause of the lowered levels of student satisfaction. 

Another problem in identifying the best organizational structure is that this evaluation worked 
with a sample size of five, in terms of variations in the organizational model. Because of such a 
small sample size, it is impossible to identify what differences in program structure lead to greater 
success. 

Key Issues and Recommendations Regarding 
the Development of SAPs in California 

SWRL believes that there are several key issues, that if successfully faced, will strengthen the 
implementation of SAPs in California. These issues and observations are, in part, based on the 
findings of this report. However, they also are based on SWRL's involvement in the SAP 
movement in California over the past few years. These issues focus on what concrete actions a 
local school district could take to support the effective development of a SAP in their district. In 
this section, SWRL discuses these key issues, and provides some specific recommendations that 
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local school districts could adopt to best address these issues in their SAP. 



Issue #1: There appear to be several characteristics of the SAPs that 

contribute to their success. A SAP, incorporating these 
characteristics, is fikely to be successful whatever its 
specific organizational form. 



SWRL believes that most any SAP program that incorporates the following characteristics will 
succeed. Instead of focusing on what kind of organizational structure to adopt, or whether to 
implement a "counselor-based" or "core-team" model, the most effective implementation may be 
the one that simply works to develop the following list of characteristics, however the SAP is 
finally organized. 

SWRL Recommendations: 

a. The SAP should develop a configuration of staff and services that is congruent with the 
social and political features of the program setting. 

b. Students referred to the SAP receive services that are appropriate for their needs. 

c. The SAP should provide an easy mechanism for students to seek help by self-referral 

d. The SAP should be able to identify and provide the most help to students who have the 
greatest need for services. 

e. The SAP should offer a range of services, including group and individual counseling. 

f . The SAP should be known as a program that maintains confidentiality, has a 
knowledgeable and helpful staff, and conducts interactions with students based on 
mutual trust, respect, and caring. 



Issue #2: For a SAP to work, there must be a philosophical 

perspective and commitment on the part of the district to 
operate the SAP as an integral part of their primary mission, 
not as an auxiliary elective. 



In most communities, the school district's primary mission is to prepare students to be functional 
young adults through their academic development — reading, writing, math, communication, etc. 
In addition to classroom academics, successful transition into young adulthood includes 
development of the student's self-discipline and self-reliance. An effective SAP serves this 
primary mission. 

In this evaluation, successful SAPs were located in districts where school boards, 
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administration, staff, parents, and taxpayers understood that the school site is an appropriate and 
logical place to address student concerns. Further, it was assumed that if these concerns went 
unaddressed, this could impede the district's mission. That is, these districts believed that until 
student needs were addressed, the student could not fully succeed academically. 

At the demonstration projects, when this commitment was weak, implementation of the SAP 
was hampered. As noted in Section 2, this was a problem in both San Dieguito and Mariposa. In 
San Dieguito, some school-site counselors specifically told the SAP Director that they did not share 
this commitment to addressing the students 1 needs; this caused some implementation problems at 
one school-site. In Mariposa, primarily because of ongoing political problems, support for the 
program at the school-site and district was \v°,ak and ultimately frustrated full implementation. 

SWRL Recommendation: 

1) When implementing a SAP, the district and school-site should formally 

incorporate into their mission statement the idea that the school is a legitimate 
and logical place to address the social and psychological needs of the students. 
Formally including this commitment into the mission statement is likely to promote a 
healthy debate about whether the school-site should involve itself in the student's life, 
beyond that of serving academic needs. This debate is essential. The objections and 
resistance that surface must be fairly and respectfully addressed. Unless everyone's 
concerns are answered, there will be persistent problems with implementai-on. Once a 
consensus is reached supporting the legitimacy of the SAP, implementation will be much 
more successful. 

Issue #3: In many ways, conducting a SAP is incompatible with many 

aspects of a school ! s functioning. For a SAP to flourish, 
ways must be found to minimize this incompatibility. 

Schools are organized, and school buildings are designed, to provide educational services to large 
groups of children, inside large classrooms, within specific time periods of the day. Schools are 
not organized to allow much individual flexibility, either in terms of how or when student learning 
is to occur. 

SAP services, however, are organized around services to individuals; at most, only small 
groups of students are served at any one time. Further, to maximize the effectiveness of the SAP 
service providers, services need to be conducted throughout the school day, not just before or after 
school, or during lunch. This results in the necessity of "pull-out " programs, where SAP services 
are delivered to students who leave their regular classroom. This causes some problems for the 
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regular classroom teacher. 

Also, because schools are designed to serve students grouped into classes of about 30, the 
physical plant of the school-site consists mainly of rooms capable of holding a large group of 
students at one time. In contrast to other organizations that provide individually based services 
(e.g., a hospital, lawyer's office), the average class room size is much larger and is inefficient for 
providing counseling services. This often leads to a shortage of appropriate room space needed to 
provide SAP services. 

In short, a number of conflicts that exist between how schools are organized and how S APs 
are organized. These conflicts exist around scheduling and providing the necessary physical 
resources to support the SAP. 

SWRL Recommendation: 

1) As part of the initial SAP implementation process, potential conflicts between 
the SAP and regular school activities should be identified. Having these issues out 
in the open and fully understood by everyone will help in two ways. First, if the issues are 
not systematically addressed in the beginning stages of implementation, the issues will be 
repeatedly encountered throughout implementation, causing unnecessary stress. As each 
phase of the SAP is implemented (e.g., initiating group services), the "battle" will have to 
be fought all over again. Making the fundamental issues apparent early on, and deciding 
on an overall general framework for solving the conflicts, will smooth implementation of 
the SAP. Unfortunately, SWRL has observed several SAPs (not the demonstration 
programs) where this problem has become a handicap. Because of the move toward 
school-based social services, even without the implementation of large numbers of SAPs, 
this problem will be encountered with increasing frequency throughout California. 

Issue #4: No matter how services are officially described, the 

students participating in the SAPs often are receiving 
counseling services. Recognizing this could help avoid 
potentially serious problems for SAPs. 

Whatever the label that is applied, SAPs are conducing both group and individual counseling on 
school campuses. At times they are dealing with very sensitive issues, such as family violence, 
serious depression, and sexuality. Students value confidentiality in SAP services, a sense of 
warmth and caring, and the ability to express their feelings in a safe place. All of these are 
essential characteristics of the counseling process. SWRL recognizes that school districts do not 
want to label SAP services as counseling because of liability and cost issues. However, that does 
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not change the objective fact that they are providing counseling services to the majority of the 
students participating in the SAP. 

A failure to recognize the SAP services for what they are creates potential difficulties. For 
example, it is inevitable that not all student outcomes will be positive. Over a five-year period in 
California, some students will commit suicide, some will engage in violence, and others will drop 
out of school because of their personal or family problems. In these kinds of situations, distraught 
parents have been known to blame almost anyone for their child's failure. The SAP could become 
an easy target for a lawsuit. It would be easy to prove that a SAP was providing de facto 
counseling services to a student who suffered some negative outcome. Technically, since there 
would be no official acknowledgment that the district was supposed to be providing those services, 
the SAP would have far overstepped its appropriate service boundaries. Today in schools, the 
school nurse cannot even supply an aspirin without permission. The consequences of providing 
counseling services — which are most often regarded as another kind of medical procedure — 
without an appropriate legal and regulatory structure are serious. 

By continuing to not acknowledge counseling services as being part of the SAP (i.e., by 
insisting that only assessment and referral occur), there also is the risk that S APs will not receive 
the funding they deserve, that personnel will not receive sufficient or appropriate training, and that 
S APs will, in general, always be "last in line" when it comes to the school budget. S APs will only 
be appropriately supported when everyone recognizes clearly what S APs are actually doing. SAPs 
are doing themselves a disservice by mislabeling their activities. 

SWRL Recommendations: 

1) The problems outlined above are too large for any one district to fully solve. Until 
regulatory clarification emerges regarding the status of SAPs, SWRL recommends that 
school districts take the following measures to avoid potential problems: 

a. Keep appropriate records, especially in cases with potentially serious problems or 
complications. 

b. Maintain appropriate supervision services for all counseling interns, students, or other 
para-professionals providing counseling services at the SAP. If appropriate 
supervision cannot be provided, serious consideration should be given as to whether 
the service should be provided at all. 

c. Develop safeguards to ensure that all reporting requirements to child abuse agencies are 
clearly and completely fulfilled. 

d. If possible, maintain close contacts with various support resources, such as 
psychologists, social workers, physicians, and others who could provide specialized 
services when needed. 
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Issue #5: There is substantial variability in the professional training of 

those serving in SAPs. For SAPs to be best implemented, 
an effective set of standards for SAP personnel should be 
established. 

SWRL observed significant variation in the demonstration sites as to who provided services for 
students. In some cases, it was graduate-level interns. In other cases, it was credentialed school 
counselors; in other cases it was talented individuals without graduate education. 

SWRL believes all of the personnel working at the demonstration sites were committed, well- 
trained, and adequate to their tasks. However, there are concerns about future SAP personnel 
should SAPs continue to expand throughout California. It is likely that, left without guidance, 
some districts will select personnel that are inadequately trained for their position. Particularly with 
significant budget difficulties, it becomes tempting to skimp on personnel costs, especially with a 
program that has no specific personnel standards or credential requirements. 

SWRL Recommendations: 

1) SWRL believes that SAP programs should develop their own internal training and/or 
educational standards for all SAP personnel within their district who deal with students. 

2) The district developed standards should provide SAP personnel guidance regarding 
minimum training requirements, continuing education, and the use of paraprofessionals, 
graduate interns, and other individuals not formally trained in the behavioral sciences. 

Issue #6: Especially when a SAP is initiated, demand for services is 

likely to be heavy. In particular, students will be likely to 
self-refer themselves for personal problems that are 
serious. The SAP should be prepared for rapid start-up. 

In this evaluation there was a heavy demand for services when the SAPs were initiated. This 
demand caught many administrators, and some of the SAP Directors, by surprise. The initial 
heavy demand placed some strain on the SAPs. In their first year they had to cope with fine-tuning 
their program operations at the same time that they needed to provide an unexpectedly high level of 
services. In retrospect, it is logical that at those sites where there were few other social service 
options, there would be a great deal of pent-up demand which would be unleashed with the SAP 
start-up. 

There was also the unexpected finding that the students who had the most serious personal 
problems tended to self refer into the SAPs. The students who had the more minor problems 
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tended to be referred through other sources. In many ways this is gratifying to see. It indicates 
that the students placed a high level of trust in the operation of the programs, a finding consistent 
with the student ratings reported in Section 4. This finding also suggests that the traditional SAP 
emphasis of developing and implementing an elaborate staff-based identification and referral 
system is not as critical as believed. Because the students with the serious problems tended to self- 
refer, marketing SAP services directly to the students might be an effective mechanism for 
attracting students into the SAP. However, this does not mean that no staff-based identification 
and referral system is necessary. There will always be students who are in need of services, but, 
for one reason or another, do not find their own way into the SAP. 

SWRL Recommendations: 

1) Staff and administrators involved in start-up of the SAP should understand that there is 
likely to be a substantial, pent-up demand for services when the SAP begins operations. 
Being prepared for a large initial demand for services will help smooth implementation. 

2) Because it was found that students with serious problems tended to self-refeyhemselves 
into the SAP, SWRL believes that the typical SAP could effectively market SAP services 
directly to students. This reduces, but does not eliminate, the need for the development and 
implementation of a staff-based identification and referral system. 
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